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The methodology developed for the Western Cooperative 
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to a Degree" projects funded by the Annenberg/CPB Project is 
described. This methodology provides an approach for conducting a 
valid, reliable evaluation effort that is sensitive to specific 
institutional issues and contingencies, while providing general izabl e 
results for cross-institutional discussions. Also included is a 
description of an environmental scan process that reveals the data 
points found most useful in evaluating distance learning projects. 
Seven projects were funded through the New Pathways initiative. The 
key to consistency in data collection and reporting within and across 
sites in this evaluation was to establish a framework within which 
the evaluations would occur. The evaluation approach used protocols 
for quantitative and qualitative investigation that facilitated 
systematic decision making with regard to: (1) context evaluation; 
(2) input evaluation; (3) process evaluation; and (A) product 
evaluation. Nine categories of information were identified as areas 
in which evaluative data were likely to be collected. The 
environmental scan provided evaluators with a way to determine the 
information actually being collected. Thirteen attachments illustrate 
the evaluation approach. (SLD) 
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Evaluating Distance Learning 
Projects: An Approach for 
Cross-Project Comparisons 

Ellen D. Wagner 

Western Cooperative for Educational 

Telecommunications - WICHE 
A presentation at the Association for Educational 
Communications and Technology Conference, 
January, 1993, based on a paper appearing in the 
Proceedings of the 8th Annual 
Conference on Distance Teaching and 
Learning, University of Wisconsin, 8/92. 

This paper will describe the inethodology 
developed for the Western Cooperative for 
Educational Telecommunications' 
evaluation of the Annenberg/CPB "New 
Pathways to a Degree" funded projects. 
This methodology provides an approach for 
conducting a valid, reliable evaluation effort 
which is sensitive to specific institutional 
issues and contingencies while also 
providing generalizable results for cross- 
institutional discussions. This paper 
include a description of an "environmental 
scan" process which revealed the data 
points described by the staff of the "New 
Pathways" funded projects as being most 
useful for evaluating distance learning 
projects. 

The New Pathways to a Degree projects 
were funded by the Annenberg/CPB 
Project to provide a means of systematically 
assessing techniques and strategies for 
improving access to undergraduate degree 
courses and programs. This has been 
accomplished in large part by using a 
variety of telecommunications technologies. 
Seven projects were funded through the 
New Pathways initiative, involving 27 
colleges and universities and three state 
networks. Funded projects include: 

• The Community College of Maine 

• Rochester Institute of Technology 

• University of West Virginia System 

• Oregon ED-NET 

• St. Catherine's College 

• IUPUI (Indiana University-Purdue 
University, Indianapolis) 

• Northern Virginia Community College 



In addition to funding these seven distance 
learning initiatives, the Annenberg /CPB 
Project also funded an independent external 
evaluation effort to be conducted by the 
Western Cooperative for Educational 
Telecommunications. The Western 
Cooperative, a program of the Western 
Interstate Commission for Higher 
Education, recently completed the first year 
of a two year evaluation effort. This paper 
will provide an overview of the processes 
used to undertake this evaluation and will 
discuss preliminary finding of the process 
undertaken to identify potential point for 
cross-project comparability. 

One of the primary purposes of the 
evaluation was to examine data within and 
across projects to answer three questions: 

•What are the effects of technology upon 
student achievement in distance learning 
contexts? 

• What is/are the most effective mix(es) of 
technologies for delivering college level 
courses, programs and services to 
diverse student audiences at a distance? 

• What data exist for determining the 
cost/benefit variables affecting 
programs delivered by technological 
means to diverse student audiences? 

Even though outcomes dealing with student 
achievement, technology mix and 
cost/benefit tend to be of greatest general 
interest when examining distance 
educational programs, the determination of 
achievement measures, technology 
effectiveness or cost/benefit appears to 
depend upon enabling conditions and 
contingencies. An additional outcome of 
this external evaluation effort was to 
determine the appropriateness of asking 
these three questions at various points in 
the life of a distance learning project. It has 
been unclear if there are stages of project 
development which are common to specific 
categories of projects, or of there are 
developmental stages common to all 
distance learning projects. The New 
Pathways evaluation provided an 
opportunity to ascertain of there is any 



discernable pattern to the stages of distance 
learning program development. 

Clearly, it was critical to develop an 
evaluation strategy which was sensitive to 
individual project constraints, 
opportunities and limitations while robust 
enough to accommodate cross-project data 
comparisons. This would help establish 
parameters for developing generalizable 
guiding principles for use with projects 
using similar technologies, working with 
similar audiences and/or under comparable 
circumstances. Ultimately this was also 
seen as a way to promote transfer, 
diffusion and dissemination of results to be 
used by others who are considering the 
initiation of similar distance learning 
efforts. 

The key to consistency in data collection 
and reporting of results within and across 
sites in this evaluation was to establish a 
framework within which the evaluations — 
within and across projects —would occur. 
The Western Cooperative for Educational 
Telecommunications' Technology 
Integration Evaluation Protocols, based 
upon Daniel Stuffelbeam's Context /Input / 
Process / Product evaluation model, 
provided such a framework. These 
Evaluation Protocols demonstrate the 
relationships among four evaluation arenas. 
This approach facilitates systematic 
decision-making as it relates to: 

• Context evaluation - This establishes the 
organizational environment and 
addresses the process of making 
organizational decisions affecting the 
project This primarily employs 
descriptive or historical information 
sources, and actively addresses the 
development of planning decisions: 
change strategies, leadership strategies, 
integration strategies, collaborative 
strategies and intra/inter-institutional 
partnerships; 

• Input evaluation - This identifies and 
examines the resources available for the 
project. This employs descriptive 
frequencies, gap analyses and 
forecasting, and actively addresses 



resource procurement, resource 
dedication, resource sharing/economies 
of scale, tactical and strategic planning 
for resource procurement; 

• Process evaluation - This identifies and 
examines the efficacy of procedures, 
operations, administration and 
management within and among 
projects. This employs qualitative 
methods of structured interviews and 
structured observations, and actively 
promotes implementation decisions; 

• Product evaluation - This identifies and 
examines outcomes, decisions to 
recycle the project(s) and judgement 
upon the advisability of project 
continuance. This employs qualitative 
and quantitative methods for descriptive 
and inferential reporting, and deals with 
decisions to modify, revise, refocus, or 
terminate a specific approach, 
intervention of and entire application. 

These Protocols were designed to utilize 
both qualitative and quantitative methods 
that are appropriate for diverse instructional 
projects employing technology for 
instructional delivery, and provide a sound 
foundation for decision-makers to make 
informed choices about the use of 
technology in their own content- and 
context-specific situations. 

In addition to establishing a conceptual 
model for guiding the selection of data 
sources, the Western Cooperative's 
Protocols specified the process and means 
through which the evaluation was to be 
conducted. This included the selection of a 
team of experts experienced in both 
evaluation methods and with distance 
learning program planning and 
implementation. Each evaluator has been 
assigned to a specific New Pathways 
funded project for two years. Each 
evaluator has been expected to conduct a 
site visit per year for the two year's 
duration of the New Pathways initiative to 
conduct extensive interviews with project 
participants. Evaluators are also expected to 
work with their project directors and 
evaluators and project staff to obtain 
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documents, reports, and other data 
specified in the Evaluation Protocols. 
Ongoing and effective communication 
among the members of the evaluation team 
has been facilitated throughout the life of 
the evaluation project by means of 
teleconferencing, electronic mail and 
occasional face-to-face meetings. 
Evaluators also use common evaluation 
instruments and consistent operational 
guidelines. An evaluation coordinator is 
responsible for directing the efforts of the 
evaluation team and for compiling and 
editing the interim and final evaluation 
reports to each funded project and the 
Annenberg/CPB Project. 

Each of the New Pathways funded projects 
included its own internal evaluation. These 
mean to ensure that the unique situational, 
institutional and environmental needs of the 
organizations receiving New Pathways 
funding have been met. However, having 
the internal evaluations evolve as a function 
of each individual project's needs meant 
that no assumptions could be made about 
comparability across projects. 

The Western Cooperative's Protocols called 
for conducting an environmental scan of all 
New Pathways projects to determine what 
data were being (or were going to be) 
collected per project, and to determine the 
relative importance of a wide array of 
potential data points on a per project basis. 
An assessment instrument was developed 
to survey each funded project's staff about: 

• the varieties and types of information 
currently being used within an 
institution to evaluate the effectiveness 
of instructional initiatives; 

• the type of information to be collected 
for the New Pathways evaluation; 

• whether or not the collected information 
could be reported in New Pathways 
evaluation reports 

• whether or not forms were currently 
available to collect the desired 
information; 



• the perception on the part of the project 
staff about the relative importance of the 
various data points included in the 
environmental scan to their own 
project's evaluation. 

Nine categories of information were 
identified as arenas within which evaluative 
data were likely to be collected. They 
included institutional characteristics; faculty 
characteristics; student characteristics; 
student psychographics; student 
achievement; course development; faculty 
development; support services; and 
technology mix. Within these nine 
categories a total of 1 17 individual data 
sources were identified. These data 
sources were presented as an exhaustive 
(albeit not exclusive) enumeration of 
commonly cited sources of information for 
evaluating distance learning projects. This 
list was generated with the input of: 

• New Pathways Associates (data 
collected during the first New Pathways 
Associates meeting held in Alexandria, 
Virginia in late April, 1991); 

• Annenberg/CPB staff involved with the 
New Pathways projects; 

• Members of the Western Cooperative's 
New Pathways evaluation team; and 

• A review of literature of technology- 
based instructional projects. 

Project directors were asked to confer with 
their project teams to complete the 
environmental scan. Each project director 
was expected to indicate the varieties, 
sources, accessibility and perceived value 
of the data being collected for use in their 
own project evaluations. With this 
information it would then be possible to 
determine whether or not cross-project 
comparisons on a per variable basis was 
remotely feasible. 

Five of the seven funded projects 
completed and returned their evaluation 
forms by the requested deadline. The two 
remaining projects, both being state 
networks, reported that the data sources 



were hard to identify because of the multi- 
institutional nature of their projects. 

The responses to the environmental scan 
provided evaluators with a means of 
determining the types of information 
actually being collected for evaluation 
purposes for each of the projects. It also 
provided a means of determining the types 
of information that were perceived to be of 
greatest value by the project directors. 

Project directors had been asked to rank the 
relative importance of the 1 17 data sources 
noted on the environmental scan for 
evaluating their New Pathways project 
from their own project perspective. 
Responses were recorded using a Likert- 
styled five point scale, with "5" noted as 
"most important" and 1 noted as "least 
important". Tabulation of results of the 
most important data points was 
accomplished by recoding the "most 
important" category to include points 4 and 
5 on the ranking scale. Recoding was also 
undertaken when tabulating the data 
sources ranked as least important by 
combining responses ranked with a "1" or a 
"2". 

The twelve categories of information wjiich 
received the highest overall ranking (that is, 
receiving either a 4 or a 5 ranking from 
each of the five institutions) for their value 
in conducting distance learning project 
evaluations are noted below. They include: 

• Profile of population served; 

• Number of students served; 

• Student motivation for pursuing distant 
learning experiences; 

• Course completion; 

• Course satisfaction; 

• Media services for faculty; 

• Media services fcr students; 

• On-line library services; 

• Cost of course delivery for technology 
based courses; 

• Computer networks used; 

• Pre/post course attitudes of students; 

• Pre/post course attitudes of faculty. 

Project staff were interviewed about their 
preferred evaluative <Jata point selections, 
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particularly since there were virtually no 
indicators through which student 
achievement could be determined in the 
"most important category. The general 
consensus was that while student 
achievement is one of the important 
determinant for persuading an institution to 
initiate a distance learning initiative, it is 
even more important to first have a 
technology system in place (including staff 
support, library resources and media 
support for faculty and student) which is 
reliable and capable of meeting student 
needs and demands. It is also critical that 
sufficient numbers of students are being 
served, and that the students receiving 
courses at a distance are satisfied with the 
experience, that the distance learning option 
compares favorably to the traditional, face- 
to-face option from a marketing 
perspective. Only after a distance delivery 
system is in place and there are sufficient' 
numbers of satisfied students receiving 
instruction at a distance are institutions able 
to devote significant energy toward 
determining student achievement outcomes 
in distance learning experiences. 

When asked about cost /benefit variables, 
project directors noted that while it is 
critical to show comparability of cost 
between tradition and technologically 
delivered courses and program, the means 
through which those determinations can be 
made provide a fair amount of leeway in 
making those cost determinations. For 
example, the heavy capital outlay for 
technology can be offset through reported 
saving in travel, in in facilities costs 
savings or of having access to additional 
sources of revenue from ad hoc system 
conferences. Project directors did note that 
the intangible benefit of providing increased 
access to instructional and support 
resources made some of their cost-benefit 
calculations moot over time. In many 
cases, the applications to which systems 
were put to use had not been anticipated in 
initial cost calculations. There was general 
consensus that once a technology system 
was in place in an institution it had 
incalculable impact on organizational 
development and institutional culture. 
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